{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recommender Systems 2020/2021\n",
    "\n",
    "## Practice Session 11 - MF with PyTorch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Outline\n",
    "\n",
    "* Dataset loading\n",
    "* Main ideas for MF\n",
    "* Model Creation\n",
    "* Dataset Loading\n",
    "* Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Movielens10M: Verifying data consistency...\n",
      "Movielens10M: Verifying data consistency... Passed!\n",
      "DataReader: current dataset is: <class 'Data_manager.Dataset.Dataset'>\n",
      "\tNumber of items: 10681\n",
      "\tNumber of users: 69878\n",
      "\tNumber of interactions in URM_all: 10000054\n",
      "\tValue range in URM_all: 0.50-5.00\n",
      "\tInteraction density: 1.34E-02\n",
      "\tInteractions per user:\n",
      "\t\t Min: 2.00E+01\n",
      "\t\t Avg: 1.43E+02\n",
      "\t\t Max: 7.36E+03\n",
      "\tInteractions per item:\n",
      "\t\t Min: 0.00E+00\n",
      "\t\t Avg: 9.36E+02\n",
      "\t\t Max: 3.49E+04\n",
      "\tGini Index: 0.57\n",
      "\n",
      "\tICM name: ICM_genres, Value range: 1.00 / 1.00, Num features: 20, feature occurrences: 21564, density 1.01E-01\n",
      "\tICM name: ICM_tags, Value range: 1.00 / 69.00, Num features: 10217, feature occurrences: 108563, density 9.95E-04\n",
      "\tICM name: ICM_all, Value range: 1.00 / 69.00, Num features: 10237, feature occurrences: 130127, density 1.19E-03\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from Notebooks_utils.data_splitter import train_test_holdout\n",
    "from Data_manager.Movielens.Movielens10MReader import Movielens10MReader\n",
    "\n",
    "data_reader = Movielens10MReader()\n",
    "datasets_dict = data_reader.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (0, 0)\t5.0\n",
      "  (0, 1)\t5.0\n",
      "  (0, 2)\t5.0\n",
      "  (0, 3)\t5.0\n",
      "  (0, 4)\t5.0\n",
      "  (0, 5)\t5.0\n",
      "  (0, 6)\t5.0\n",
      "  (0, 7)\t5.0\n",
      "  (0, 8)\t5.0\n",
      "  (0, 9)\t5.0\n",
      "  (0, 10)\t5.0\n",
      "  (0, 11)\t5.0\n",
      "  (0, 12)\t5.0\n",
      "  (0, 13)\t5.0\n",
      "  (0, 14)\t5.0\n",
      "  (0, 15)\t5.0\n",
      "  (0, 16)\t5.0\n",
      "  (0, 17)\t5.0\n",
      "  (0, 18)\t5.0\n",
      "  (0, 19)\t5.0\n",
      "  (0, 20)\t5.0\n",
      "  (0, 21)\t5.0\n",
      "  (1, 16)\t3.0\n",
      "  (1, 22)\t5.0\n",
      "  (1, 23)\t3.0\n",
      "  :\t:\n",
      "  (69877, 463)\t3.0\n",
      "  (69877, 467)\t1.0\n",
      "  (69877, 468)\t4.0\n",
      "  (69877, 475)\t2.0\n",
      "  (69877, 481)\t3.0\n",
      "  (69877, 486)\t4.0\n",
      "  (69877, 505)\t3.0\n",
      "  (69877, 518)\t1.0\n",
      "  (69877, 537)\t5.0\n",
      "  (69877, 541)\t2.0\n",
      "  (69877, 1081)\t2.0\n",
      "  (69877, 1302)\t4.0\n",
      "  (69877, 1322)\t2.0\n",
      "  (69877, 1436)\t4.0\n",
      "  (69877, 1609)\t1.0\n",
      "  (69877, 1646)\t3.0\n",
      "  (69877, 1660)\t2.0\n",
      "  (69877, 1671)\t2.0\n",
      "  (69877, 2001)\t4.0\n",
      "  (69877, 2065)\t1.0\n",
      "  (69877, 2941)\t1.0\n",
      "  (69877, 3066)\t1.0\n",
      "  (69877, 3386)\t3.0\n",
      "  (69877, 3448)\t1.0\n",
      "  (69877, 5330)\t1.0\n"
     ]
    }
   ],
   "source": [
    "URM_all = datasets_dict.AVAILABLE_URM[\"URM_all\"]\n",
    "print(URM_all)\n",
    "\n",
    "URM_train, URM_test = train_test_holdout(URM_all, train_perc = 0.8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MF models rely upon latent factors for users and items which are called 'embeddings'\n",
    "\n",
    "![latent factors](https://miro.medium.com/max/988/1*tiF4e4Y-wVH732_6TbJVmQ.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_factors = 10\n",
    "\n",
    "n_users, n_items = URM_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "# Creates U\n",
    "user_factors = torch.nn.Embedding(num_embeddings=n_users, embedding_dim=num_factors)\n",
    "\n",
    "# Creates V\n",
    "item_factors = torch.nn.Embedding(num_embeddings=n_items, embedding_dim=num_factors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Embedding(69878, 10)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_factors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Embedding(10681, 10)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item_factors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### To compute the prediction we have to multiply the user factors to the item factors, which is a linear operation.\n",
    "\n",
    "### We define a single layer and an activation function, which takes the result and transforms it in the final prediction. The activation function can be used to restrict the predicted values (e.g., sigmoid is between 0 and 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the [nn.Linear docs](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html?highlight=linear#torch.nn.Linear)\n",
    "\n",
    "Applies a linear transformation to the incoming data: $$y = xA^T + b$$\n",
    "\n",
    "In our case, it will transform the element-wise multiplication of `user_factors` and `item_factors` into a rating prediction.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Linear(in_features=10, out_features=1, bias=True)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "layer_1 = torch.nn.Linear(in_features=num_factors, out_features=1)\n",
    "layer_1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the [ReLU docs](https://pytorch.org/docs/stable/generated/torch.nn.ReLU.html#torch.nn.ReLU)\n",
    "\n",
    "$$ ReLU(x) = max(0,x) $$\n",
    "\n",
    "![image](https://pytorch.org/docs/stable/_images/ReLU.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ReLU()"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "activation_function = torch.nn.ReLU()\n",
    "activation_function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## In order to compute the prediction you have to:\n",
    "1. Define a list of user and item indices\n",
    "2. Create a tensor from it\n",
    "4. Get the user and item embedding\n",
    "5. Compute the element-wise product of the embeddings\n",
    "6. Pass the element-wise product to the single layer network\n",
    "7. Pass the output of the single layer network to the activation function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.  [42] <class 'list'>\n",
      "2.  tensor([42]) <class 'torch.Tensor'>\n",
      "3.  tensor([[-0.0720,  0.5358, -0.8721, -1.7628,  0.6615, -1.7984, -0.9140, -0.8624,\n",
      "          0.6267,  1.2498]], grad_fn=<EmbeddingBackward>) <class 'torch.Tensor'>\n",
      "4.  tensor([[ 0.0408,  0.3476, -0.0644, -3.2816,  0.5467,  0.9337, -0.7592, -0.9672,\n",
      "          1.0566,  0.4944]], grad_fn=<MulBackward0>) <class 'torch.Tensor'>\n",
      "5.  tensor([[-0.2182]], grad_fn=<AddmmBackward>) <class 'torch.Tensor'>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "('6. ', tensor([[0.]], grad_fn=<ReluBackward0>), torch.Tensor)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 1. Define a list of user/item indices.\n",
    "item_index = [15]\n",
    "user_index = [42]\n",
    "print(\"1. \", user_index, type(user_index))\n",
    "\n",
    "# 2. Create a tensor from it. Specify indices are of type int64.\n",
    "user_index = torch.Tensor(user_index).type(torch.int64)\n",
    "item_index = torch.Tensor(item_index).type(torch.int64)\n",
    "print(\"2. \", user_index, type(user_index))\n",
    "\n",
    "# 3. Get the user and item embeddings \n",
    "current_user_factors = user_factors(user_index)\n",
    "current_item_factors = item_factors(item_index)\n",
    "print(\"3. \", current_user_factors, type(current_user_factors))\n",
    "\n",
    "# 4. Compute the element-wise product of the embeddings\n",
    "element_wise_product = torch.mul(current_user_factors, current_item_factors)\n",
    "print(\"4. \", element_wise_product, type(element_wise_product))\n",
    "\n",
    "# 5. Pass the element-wise product of the embeddings\n",
    "prediction = layer_1(element_wise_product)\n",
    "print(\"5. \", prediction, type(prediction))\n",
    "\n",
    "# 6. Pass the output of the single layer network to the activation function\n",
    "prediction = activation_function(prediction)\n",
    "\"6. \", prediction, type(prediction)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### To take the result of the prediction and transform it into a traditional numpy array you have to first call .detach() and then .numpy()\n",
    "### The result is an array of 1 cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction is [[0.]]\n"
     ]
    }
   ],
   "source": [
    "prediction_numpy = prediction.detach().numpy()\n",
    "print(\"Prediction is {}\".format(prediction_numpy))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train a MF MSE model with PyTorch\n",
    "\n",
    "# Step 1 Create a Model python object\n",
    "\n",
    "### The model should implement the forward function which computes the prediction as we did before"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MF_MSE_PyTorch_model(torch.nn.Module):\n",
    "    def __init__(self, n_users: int, n_items: int, n_factors: int):\n",
    "        super(MF_MSE_PyTorch_model, self).__init__()\n",
    "\n",
    "        self.n_users = n_users\n",
    "        self.n_items = n_items\n",
    "        self.n_factors = n_factors\n",
    "\n",
    "        self.user_factors = torch.nn.Embedding(num_embeddings=self.n_users, embedding_dim=self.n_factors)\n",
    "        self.item_factors = torch.nn.Embedding(num_embeddings=self.n_items, embedding_dim=self.n_factors)\n",
    "\n",
    "        self.layer_1 = torch.nn.Linear(in_features=self.n_factors, out_features=1)\n",
    "        self.activation_function = torch.nn.ReLU()\n",
    "\n",
    "    def forward(self, user_coordinates, item_coordinates):\n",
    "        current_user_factors = self.user_factors(user_coordinates)\n",
    "        current_item_factors = self.item_factors(item_coordinates)\n",
    "\n",
    "        prediction = torch.mul(current_user_factors, current_item_factors)\n",
    "        prediction = self.layer_1(prediction)\n",
    "        prediction = self.activation_function(prediction)\n",
    "\n",
    "        return prediction\n",
    "\n",
    "    def get_W(self):\n",
    "\n",
    "        return self.user_factors.weight.detach().cpu().numpy()\n",
    "\n",
    "    def get_H(self):\n",
    "\n",
    "        return self.item_factors.weight.detach().cpu().numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 2 Setup PyTorch devices and Data Reader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_MSE_PyTorch: Using CPU\n"
     ]
    }
   ],
   "source": [
    "if torch.cuda.is_available():\n",
    "    device = torch.device('cuda')\n",
    "    print(\"MF_MSE_PyTorch: Using CUDA\")\n",
    "else:\n",
    "    device = torch.device('cpu')\n",
    "    print(\"MF_MSE_PyTorch: Using CPU\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create an instance of the model and specify the device it should run on"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "pyTorchModel = MF_MSE_PyTorch_model(n_users, n_items, num_factors).to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Choose loss functions (Mean Squared Error in our case), there are quite a few to choose from"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "lossFunction = torch.nn.MSELoss(reduction=\"sum\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Select the optimizer to be used for the model parameters: Adam, AdaGrad, RMSProp etc... "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "learning_rate = 1e-4\n",
    "\n",
    "optimizer = torch.optim.Adagrad(pyTorchModel.parameters(), lr=learning_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define the DatasetIterator, which will be used to iterate over the data\n",
    "\n",
    "### A DatasetIterator will implement the Dataset class and provide the __getitem__(self, index) method, which allows to get the data points indexed by that index.\n",
    "\n",
    "### Since we need the data to be a tensor, we pre inizialize everything as a tensor. In practice we save the URM in coordinate format (user, item, rating)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import Dataset\n",
    "import numpy as np\n",
    "\n",
    "class DatasetIterator_URM(Dataset):\n",
    "    def __init__(self, URM):\n",
    "        # Remember that URM[row[k], col[k]] = data[k]\n",
    "        URM = URM.tocoo()\n",
    "\n",
    "        self.n_data_points = URM.nnz\n",
    "\n",
    "        # Create a nx2 tensor where: A[i,0] = row[i] and A[i,1] = col[i]\n",
    "        self.user_item_coordinates = np.empty((self.n_data_points, 2))\n",
    "        self.user_item_coordinates[:,0] = URM.row.copy()\n",
    "        self.user_item_coordinates[:,1] = URM.col.copy()\n",
    "        self.user_item_coordinates = torch.Tensor(self.user_item_coordinates).type(torch.int64)\n",
    "       \n",
    "        # Converts ratings to tensor.\n",
    "        self.rating = URM.data.copy().astype(np.float)\n",
    "        self.rating = torch.Tensor(self.rating) # No need to specify type as torch.Tensor by default is torch.float32\n",
    "\n",
    "    def __getitem__(self, index):\n",
    "        \"\"\"\n",
    "        Format is (row, col, data)\n",
    "        :param index:\n",
    "        :return:\n",
    "        \"\"\"\n",
    "        return self.user_item_coordinates[index, :], self.rating[index]\n",
    "\n",
    "\n",
    "    def __len__(self):\n",
    "        return self.n_data_points\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We pass the DatasetIterator to a DataLoader object which manages the use of batches and so on..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader\n",
    "\n",
    "batch_size = 200\n",
    "\n",
    "dataset_iterator = DatasetIterator_URM(URM_train)\n",
    "\n",
    "train_data_loader = DataLoader(dataset=dataset_iterator,\n",
    "                               batch_size=batch_size,\n",
    "                               shuffle=True,\n",
    "                               #num_workers = 2,\n",
    "                              )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## And now we ran the usual epoch steps\n",
    "* Data point sampling\n",
    "* Prediction computation\n",
    "* Loss function computation\n",
    "* Gradient computation\n",
    "* Update"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fd8d8b238f5a480e9b9648bd0e5121d3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=40002), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "CPU times: user 4min 18s, sys: 16.6 s, total: 4min 35s\n",
      "Wall time: 3min 12s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "from tqdm import tqdm_notebook as tqdm\n",
    "\n",
    "for input_data, ratings in tqdm(train_data_loader, 0):\n",
    "    optimizer.zero_grad()\n",
    "    cumulative_loss = 0.0\n",
    "\n",
    "    user_coordinates = input_data[:,0]\n",
    "    item_coordinates = input_data[:,1]\n",
    "\n",
    "    # FORWARD pass\n",
    "    predictions = pyTorchModel(user_coordinates, item_coordinates)\n",
    "    predictions = predictions.view(-1) # predictions are a 1xbatch tensor, we just want an array of size batch (as ratings)\n",
    "    \n",
    "    # Obtain loss score, basically the MSE of the predicted and actual ratings\n",
    "    loss = lossFunction(predictions, ratings)\n",
    "    \n",
    "    # BACKWARD pass\n",
    "    loss.backward()\n",
    "    optimizer.step()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## After the train is complete (it may take a while and many epochs), we can get the matrices in the usual numpy format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_factors = pyTorchModel.get_W()\n",
    "item_factors = pyTorchModel.get_H()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([[ 0.26520982, -0.68337   , -0.54723114, ...,  1.3686696 ,\n",
       "          0.61501884,  0.35410157],\n",
       "        [-1.6309935 ,  0.8811241 ,  1.5944738 , ..., -0.6269186 ,\n",
       "         -0.12894574,  0.4979117 ],\n",
       "        [-1.1888154 , -1.8266233 , -0.17145918, ..., -0.6406859 ,\n",
       "          0.21364413, -0.03301006],\n",
       "        ...,\n",
       "        [-0.1464834 ,  1.2325315 ,  0.21650487, ..., -1.081837  ,\n",
       "         -1.3280292 , -0.5451829 ],\n",
       "        [-0.48614633, -0.27829456,  0.23680773, ..., -0.2558126 ,\n",
       "         -0.09771301, -0.53594506],\n",
       "        [-0.30689088,  0.47045732,  1.2976024 , ..., -1.4160035 ,\n",
       "         -0.7244118 ,  2.3710787 ]], dtype=float32),\n",
       " (69878, 10))"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_factors, user_factors.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([[ 0.33532315, -1.4667886 ,  1.2425084 , ...,  1.7038391 ,\n",
       "         -0.51159924, -1.2737192 ],\n",
       "        [ 0.6346393 , -0.4209208 , -0.973261  , ..., -1.2520459 ,\n",
       "          0.7278145 , -0.15892538],\n",
       "        [ 1.1904838 ,  0.10658428,  0.5855952 , ...,  0.5420558 ,\n",
       "          0.39672065, -0.4849253 ],\n",
       "        ...,\n",
       "        [ 1.4554454 ,  1.1165615 ,  0.5367837 , ..., -0.7780073 ,\n",
       "         -0.8466227 ,  0.74578154],\n",
       "        [-0.54399824,  1.0597395 ,  0.90279496, ..., -0.55121493,\n",
       "          0.6213373 ,  0.168343  ],\n",
       "        [ 0.39889905,  0.27928987, -0.14391598, ...,  1.5336596 ,\n",
       "          0.45261514, -0.82100457]], dtype=float32),\n",
       " (10681, 10))"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item_factors, item_factors.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
